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The Rigidity and Comfort of Habits: 

A Cultural and Philosophical Analysis of the Ups and Downs of 
Mainstreaming Evaluation 



ABSTRACT 

Mainstreaming evaluation requires establishing aesthetic and ethical 
frameworks, as well as knowledge and skills, that make "doing" 
evaluation seem like the right thing. Evaluators and others have 
worked hard to institute evaluation as a prudent activity for society to 
support. The phenomenon of mainstreaming itself, however, poses 
challenges to good evaluation practice. Stories from program 
evaluation research in North America and Indonesia in the areas of 
conservation and natural resource management illustrate the ways in 
which comforts and rigidities associated with mainstreamed 
evaluation processes may frustrate high quality evaluation. The stories 
are couched in neo-pragmatist analysis of the phenomenon of habits 
(i.e., settled knowledge, cultural ethos) using theories developed by 
Kurt Lewin, John Dewey, and Richard Rorty on the nature of stasis 
and change. Adult education theories related to changing adults’ views 
is introduced as important to the concept of change, which mirrors the 
concept of mainstreamed in this context. A cultural anthropological 
view further suggests that evaluation can grow to have strikingly 
different meanings in other settings, complicating the idea of 
mainstreaming as a preferred steady state. 



So we must face the discomfort of stirring ourselves if we are to avoid being left behind. 

E. Chelimsky & W. R. Shadish (1997). Evaluation for the 21 s ' century. Thousand Oaks, 
CA: Sage, p. xiii. 



Introduction 

My paper argues that the concept of mainstreaming, this year's presidential strand 
topic for the American Evaluation Association annual meeting, presents certain problems. 
I do not intend to reject or endorse mainstreaming, either as a concept or as a 
phenomenon. Indeed, mainstreaming happens, and needs to happen, and it is a credit to 
dedicated and astute evaluators working in many fields. Rather, my paper offers attendees 
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an opportunity to think philosophically about the phenomenon of mainstreaming from 
what is, perhaps, a different point of view. I approach evaluation from the perspective of 
cultural anthropology, which means that I cannot avoid viewing evaluation as a cultural 
event. My arguments, therefore, offer as much an "anthropology of evaluation" as 
anything else. I am also comfortable introducing my ideas through a story, something that 
anthropologists like to do as well as anyone else. The story centers around a trip to 
Indonesia in January 2000. 

Indonesia, January 2000 

I secured funds to travel to several islands in eastern Indonesia to witness a large- 
scale participatory evaluation of an exemplary and well-funded conservation and 
development project (Fisher, Moeliono, & Wodicka, 1998). I intended to learn how 
evaluation occurred in a large-scale project that had used participatory evaluation and 
action research throughout its development (Wodicka, 1999). My excitement was fueled 
by the idea of witnessing participatory evaluation in a cultural context that was strikingly 
different from that of my North American-based work. Participatory evaluation has many 
forms but, typically, features shared control of evaluation processes. Participatory 
evaluation also intends to involve a wider range of stakeholders in collecting and 
assessing data than is usual in other approaches to evaluation (Cousins & Whitmore, 
1998). 

I interviewed staff and researchers as we toured islands where villagers raised 
corn on rainforest hillsides, fished magnificent coral reefs, and complained about 
Komodo dragons eating their goats. Everywhere was participation: participatory action 
research with villagers and government to address conflicts over forest boundaries. 
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trainers and trainees discussing participatory rural appraisal, and social forestry projects. I 
was privileged to meet participatory non governmental organization (NGO) facilitators 
and village leaders working to better the lives of people in a country that had seen many 
years of repression. As a self-professed participatory junkie, I was thrilled and wowed to 
make this trip. 

The story takes a turn here. You knew that, didn't you? 

The turnabout surprised me— my biases and assumptions tripping me up again. My 
unsettling began as I asked about the project's participatory-evaluation-in-progress during 
personal interviews and through follow-up phone interviews six months later. It seems 
there were various complaints about the evaluation not being as participatory as people 
had hoped, or as participatory as people were used to experiencing. An Indonesian 
national close to the project said, "I think that the evaluation of the consortium is not so 
participatory." A key staff person confirmed, "We just did an evaluation. We weren't the 
facilitators of it. We hired outside facilitators, but they were very open in terms of doing a 
participatory evaluation and really playing the role of facilitators rather than outside legal 
kind of evaluation and passing judgments." He later added, "the facilitators sometimes 
responded as evaluators and sometimes responded as facilitators. Sometimes they were 
really trying to influence the recommendations. They brought in their own biases, their 
own way of thinking about the network." The outsider process favored 
"recommendations rather than lessons learned." This produced difficult "arduous 
discussions" that this and other participants experienced as frustrating, particularly when 
evaluation facilitators asked for consensus on a key issue, received none, so requested a 
vote that was overturned the next day (after the facilitators had left). The upshot 
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portrayed an evaluation process that was less participatory than much of their ongoing 
evaluative work; therefore, the results were considered less beneficial and the process, 
less than ideal. 

This story is not remarkable in its disarray, or even in its flip-flop processes. 
Moreover, the story is not unique in showing that evaluation is impure, complex, and 
socially-constructed, features understood in the evaluation field (see Guba, 19XX; 

Patton, 1997; and others). At first, I presumed that the failure to employ participatory 
processes had something to do with the participatory methods themselves. Perhaps 
participation was feared as dangerously political, too revealing, or even tyrannical (see 
for example XX and Kothari, 2000). This enormous archipelago, as we are perhaps more 
aware than in years past, is the largest Muslim country in the world, and continues to 
experience ethnic and religious conflict (Chambers, 1997; Mayer, 1996). Risks associated 
with methods of inquiry certainly occur in the real world. We ignore risks at our own 
peril— more so the peril of our clients— for evaluations conducted in politically unstable 
areas of the world. I admit, however, that this preliminary and perhaps romantic analysis 
was colored by the we-would-be-martyrs syndrome that I sometimes exude as I witness 
the lukewarm reception of participatory techniques in some contexts. But I do not think 
this was the right analysis. I think a better analysis has more to do with the effect of 
mainstreaming in evaluation, and the way in which mainstreaming is also another way to 
say "enculturated." 

Culture and Evaluation 

When something is enculturated, the event or behavior has meaning that is 
intimately connected to tacit webs of understanding, to paraphrase cultural anthropologist 
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Clifford Geertz (1973). It is important that we consider that evaluation, like other events 
and behaviors, is part of culture, and developed by people to have meanings particular to 
time and place. Such meanings include yet are beyond how evaluation professionals— 
cultural creatures themselves— define themselves and their activities. We may realize that 
clients, funders, and organizations are under the spell of conventions, histories, and 
traditions. They have cultures. I submit that we avoid, however, whole-hog analysis of 
ourselves and our activities as culture-bound. We admit "biases," but these are typically 
portrayed as individual, as under own control, or as superficial. And so we are able to 
"bracket" them, as the jargon goes, and get on with the evaluation. 

Ernie House and Ken Howe (1999) have pushed this line of thinking for us 
through their work on values in evaluation. They suggest that we consider two ways of 
managing key challenges in evaluation. 

There are formal and informal ways of doing so. The formal ways take the 
form of rules and procedures . . . There is less agreement on deriving 
criteria and combining results. These procedures have not undergone 
formalization or rigorous scrutiny. In these cases, we resort mostly to 
informal reasoning, (p. X) (italics added). 

What is inside the black box of informal reasoning? I argue that "culture" is inside, and 
not just culture with a big "C." Commonly, one thinks about culture as long standing 
historical and ethnic attributes— foods, mannerisms, and customs. Some in our community 
are also aware that honoring culture involves coming to terms with power (Brisolara, 
1998; Guba and Lincoln, 1989; Rorty, 1982). In combination, these are important and 
essential dimensions of culture that press us unendingly. However, on an everyday basis. 
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even when we believe we have figured out big ‘C’ stuff, there is always more culture 
knocking at the door. Notably, local "organizational cultures" occur, in evaluation units 
as well as in client organizations. Such cultures are surprisingly common. It may be most 
practical to state that culture "writ large and small" must be considered when unpacking 
such phrases as informal reasoning. 

Eleanor Chelimsky and William Shadish (1999) acknowledge the force of culture, 
suggesting, "It is often uncomfortable to stir oneself from familiar cultural, topical, 
conceptual, and methodological niches." They end, "So we must face the discomfort of 
stirring ourselves if we are to avoid being left behind" (p. xiii). What’s underdeveloped 
among us, suggest Chelimsky and Shadish, is the acknowledgement that we are part of 
culture, and so is evaluation. I submit that not only must we avoid "being left behind," 
but must strive to avoid other nasties that result from ignoring 'our' culture as well as their 
culture, such as gathering bad data, ignoring important stakeholders, and overestimating 
peoples' willingness to assist evaluation. Such mistakes can put contribute to an 
evaluation that violates professional standards (Sanders, 1994). 

Tacit Knowledge and Habits 

This problem of not seeing culture may be inevitable. In anthropology, culture is 
talked about using an analogy based on fish and water. We say, the fish is the last 
creature to see the water to indicate the marvelous ability of people to fail to notice the 
rules that govern them (see also Bateson, 1980). In the professions, attending 
unconsciously to fundamentals isn't all bad. Part of moving into a professional role is to 
learn to do the job so well that we don't have to think about it. Donald Schon (1983) 
enlightened us to the patterned ways in which informal and intuitive professional 
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knowledge is both tacit and powerful. Personally, I like the feeling of moving seamlessly 
among tasks, zipping along, unencumbered by hesitation. The concept of mainstreaming 
can be viewed in this light as processes that evaluators and organizations take for granted. 

If we believe Schon (1983), we might further conclude that many of our 
professional ideas exist at an unconscious level and are nearly unshakable. Moreover, to 
the extent that evaluators are similar to other professionals, evaluators are unlikely to be 
able to talk about their unconscious, unshakeable ideas. There is a downside to working 
from the heart or relying on one's gut instinct. When circumstances change, trouble 
begins. For example, when faced with clear evidence that their ideas aren't working, 
Schon and his colleague Chris Argyris discovered that professionals typically dismiss the 
evidence (Argyris & Schon, 1996). Worse, the professionals that were able to accept 
negative evidence as valid routinely failed to act differently— even when they tried. This 
may occur among evaluators as well— excluding, of course, other speakers here, our fine 
president and his executives, and all of you in the audience today. 

I ask your patience as I return to the story of evaluation in Indonesia. 

The tip-off that evaluation was playing its cultural hand in the Indonesia 
evaluation setting was the lack of discussion about the pros and cons of participatory 
evaluation itself. For example, people did not talk about the risks of participatory 
evaluation, nor did people dwell on practical problems or benefits of participatory 
evaluation. Moreover, although situated in a nation very different from my own, neither 
outsiders nor insiders talked about the decision to use an outside evaluation process as 
"Indonesian" in any cultural sense. 
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Instead, people said they did not use participatory evaluation because it was the 
ten year anniversary of the program. Apparently, for ten year anniversary evaluations, a 
comprehensive, external evaluation felt right. 

Now go figure. What would you do with this response? 

Early on, I dismissed this "anniversary" explanation. It did not seem worth 
noticing. Why would a ten year anniversary cause an organization with a remarkable 
history of participatory education and evaluation to opt for an outsider evaluation process 
heavy on recommendations? Nonetheless, continued discussions bore this out. A staff 
person said, 

Of course, we were having had ten years of this consortium. We were 
interested in knowing what kind of impact we have had so far ... we 
were interested in having a kind of independent team, people from the 
outside coming to the consortium and helping us identify what lessons 
we have learned from the work that we have been doing over the last 

ten years This was more or less like a ten year anniversary, 

where we wanted to set some time aside specifically for evaluation. 

The notion of anniversary warrants unpacking. Anniversaries are complex and particular 

to contexts, but their cues are nearly always related to time, ritual, authority, and 

reconstructing the story. This is not silly stuff. It is, in fact, culture. The prospect of ritual 

related to anniversaries is powerful not just in Indonesia, but in non governmental 

organizations and in the workplace in the US. After all, we would not be lauding the 

concept of mainstreaming as a presidential strand event if we were not at least partly 

susceptible to the impetus of celebration and ritual as conducted properly by President 

Sanders. Evaluation enacts culture at the same time that evaluation is an expression of 

culture. 
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Culture and Mainstreaming 

The cultural view bears directly on the issue of mainstreaming. I can argue, for 
example, that evaluation— of whatever stripe— is mainstreamed when it becomes part of 
organizational culture. This is both good news and bad news. The good news is that once 
part of the culture, habits become regular. People enact cultures tacitly, naturally, 
intuitively. They don't have to think about it. Moreover, culture is durable, so once 
mainstreamed, evaluation sticks. 

The bad news is that once mainstreamed, it is hard to change, even when change 
is warranted. For example, people's ideas about evaluation are often regrettable. 
Evaluation can be a way for a business to downsize. Evaluation can be used to unjustly 
remove particular people or programs. Many of us complain about people in 
organizations who hate evaluation (and evaluators) and manage to sabotage inquiry. 
Sometimes, one can scratch the surface of such an organization's lack of cooperation and 
discover a legacy of accountability that harassed employees. An astute speaker at our 
conference, Kalyani Rai (2001), revealed that among her Asian immigrant clients, 
evaluation is equivalent to colonization. In these cases, resistance to evaluation rather 
than cooperation makes sense. My students are often appalled, on the other hand, by 
organizations that distribute end-of-meeting forms with mechanical precision but 
mindless attention. Evaluation, in these cases, is perfunctory. Habits comprise 
mainstreamed evaluation, but it doesn't matter. When evaluators try to change these types 
of mainstreamed habits, one learns a lot about resistance. 

Metaphors are avenues to understanding what evaluation means to different 
organizations. About eight years ago, I conducted an evaluation that clients repeatedly 




01/05/02 



10 



11 



New Directions submitted 12/01 



described as "ammunition" for their organization (Grudens-Schuck, 1998). More 
recently, a prospective client talked about evaluation as a "shield." Such metaphors are 
crucial to understanding how clients see the role of the evaluator. Chances are, 
evaluators will come and go but metaphors will remain. I wonder what examples you 
have your own from evaluation practice. 

Rigidity 

A hallmark of habits is rigidity. Again, this is not so much judgment as it is a 
practical matter. It is an oxymoron to think of irregular habits or elusive habits. Habits 
are habits because they have integrity in their sameness, are well-integrated with other 
parts of the culture, and resist change. Ideas about evaluation that are durable and 
deleterious to the progress of organizations and to public knowledge and accountability 
are regrettable. Rigidity can be problematic. 

As we examine mainstreaming, we might benefit from consideration of the sister 
concepts of interruption and change. Organizational habits can change, and thank 
goodness. Understanding change would help evaluators address several questions related 
to mainstreaming: 

How may evaluators successfully introduce hew evaluation methods? 

What ideas about evaluation do clients hold tightly or lightly? 

How may evaluators assist people to unlearn outmoded habits related to evaluation? 
How might evaluators assist organizations to institutionalize evaluation protocols in 
ways that prevent relapse, especially in the face of key staff or leadership turn-over? 
What are the limits of training, for ourselves and others, with respect to habits of 
evaluation that are tacit and rigid? 
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Remember that the flip side of this issue acknowledges mainstreaming as potentially 
healthy. Mainstreamed habits reduce thinking. Hey— I am fine with that. However, as my 
colleague Lynn Jones teaches his students and clients, evaluators are charged with 
causing thinking and learning when they enter an evaluation context (personal 
communication). Evaluation is a social intervention (Guba & Lincoln, 1989). It is a way 
to assist people to interact or process in non-trivial, non-ordinary ways (Bateson, 1980). 
Evaluation can be an opportunity to structure relationships differently so that the 
necessary conversations can occur and people can learn. 

Catalyzing Change 

I will not present a comprehensive framework for change of people and 
organizations, but I will touch upon a few key concepts. First, educational psychology 
suggests that difference and disorientation are necessary for thinking "outside the box" 
(Mezirow, 1991). The concept of unfreezing-freezing developed by social psychologist 
Kurt Lewin (1948) is instructive here. Lewin studied how people might institute new 
practices in industry and in the home. Lewin knew that habits undermine change— even 
when change made sense or was anticipated to help. Lewin coined the terms unfreezing- 
freezing to suggest that new learning had to be preceded by a period of unlearning. 

Adult educators continue to find these concepts useful. Adults who need to change 
usually know the technical aspects of change, but don't know how to make it happen. 

Lewin further discovered that learning and unlearning were not discrete, e.g., 
people did not substitute one concept for another. Rather, in the phase of unfreezing, 
concepts are loosened and muddled, hence the slushy metaphor. The contemporary term 
constructivism now enfolds such concepts (see also Guba & Lincoln, 1989). A 
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constructivist framework assumes that people create new knowledge based on prior 
knowledge. Specific, even confrontational, methods are required to get people to 
unlearn what they already know about evaluation, marriage, conferences, Aunt Pearl, 
anthrax, Toronto, or anything else. Adult educator Jack Mezirow (1991) documented the 
essential role of a disorienting dilemma in catalyzing significant learning. The mistake is 
to pour new knowledge over old and expect long lasting change. 

Pragmatism, Habits, and Mainstreaming 

John Dewey (1938), and modern neo-pragmatist philosophers like Richard Rorty 
(1982), drive this point home about the rigidity of habits by presenting habits as steady 
yet tentative. Dewey's notion of "settled knowledge" advocates for a view of behavior 
and beliefs as susceptible to change, even when they seem 'rock hard'. For Dewey, there 
was no final knowledge, no pure essence of truth. These beliefs are what make him a 
pragmatist, not in the sense of being simplistic or superficial, but as a co-founder of a 
philosophy that dismisses the search for perfect truth (see Langsdorf & Smith, 1995). 

The implications of pragmatist ideas for the concept of mainstreaming in 
evaluation are two-fold. On the one hand, evaluation can be tentative yet not seem that 
way because there is a great advantage to viewing settled knowledge as firm and 
actionable. Additionally, a pragmatist view would emphasize that mainstreamed 
evaluation practices are enmeshed in present time, situated in particular contexts. As 
contexts change, as beliefs become unsettled, what is currently mainstreamed may 
become habits that we would hope become unlearned. Of course, we live this tension in 
the American Evaluation Association. Egon Guba and Yvonna Lincoln's (1989) Fourth 
Generation Evaluation made many evaluators-none present here today-squirm and 
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become disoriented. Perhaps their strong claims accomplished Lewin's (1948) 
unfreezing or provided a Mezirowian (1991) disorienting dilemma. Participatory 
evaluation and otherinnovations may be the new habits in Lewin's refreezing phase. 

A Role for a Cultural View 

How is it helpful for evaluators to frame mainstreaming as a cultural creature? 
For one, we might listen more carefully to the ways in which organizations and clients 
talk about evaluation. Instead of simply receiving metaphors as humorous or clever, as I 
confess I often do, we might use them as a point of serious engagement. Is evaluation, 
for example, a ten year anniversary? Ammunition? A shield? Colonization? If so, what 
does that mean? An early qualitative phase of an evaluation, starting with negotiation of 
the contract (if possible), could document important terms. The terms' frequency of use, 
application by different stakeholders, and connection to actions could inform the 
evaluation (or at least, the evaluator). I am considering adding a "client metaphor" 
category to my client intake form so that I remember to document. 

Second, we might view resistance to new ideas and adherence to old (whatever 
those might be) less as positivist (or relativist) bull-headedness and more as habits in 
Lewin's frozen mode. Third, we might relax our expectation that clients and participants 
in programs readily explain why they prefer or require what seem like outmoded, 
unhelpful, or unethical evaluation processes. Over time, interpretive forms of evaluation 
can surface meanings, but this is usually not possible at the outset. 

If we decide to change evaluation practices that are currently mainstreamed (as • 
participatory evaluators try to do), or attempt to instill a new set of habits (gosh, another 
thing participatory evaluators try to do), we would do well to consider Lewin's (1948) 
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unfreezing-freezing model. Such an approach would necessitate an adult education 
framework that assisted clients to surface, then unlearn, non preferred habits while 
considering new ones. 

Fourth, we might also give in to working with the multi-faceted, culturally- 
bound purposes of our clients— if these purposes enable us to act ethically and 
effectively. (An evaluator might refuse, for example, to enhance symbolic dimensions of 
an evaluation that clients talk about as 'vengeance.'). For more benign puiposes, 
evaluators can help clients think through aesthetic or symbolic dimensions of the 
evaluation that affect cost, timing, setting objectives, and data collection. If, for 
example, clients say that evaluation must be a celebration, it would be prudent, for 
example, to ask how the report might look different from several earlier reports (e.g., 
higher quality, more copies, higher cost?). Different stakeholders might need to be 
involved. Authority figures might be more important, and results might need to be 
presented at an upcoming event that would rush data collection. One might also explore 
how negative findings would be handled in an evaluation that is intended to be a 
celebration. 

The tacit skill of "psyching out" clients is something that good evaluators already 
do, at least with the clients with which they have learned to succeed. Greater attention to 
cultural and philosophical dimensions of evaluation area might assist us to succeed with 
different types of clients, or help clients face new pressures. What is unworthy of us as 
social inquirers is to underestimate the complexity of mainstreaming, its tacit control 
over us as well as over "them." One way to surmount this challenge would be to develop 
a sense of evaluation as an artifact of culture that evaluators shape rather than originate. 
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